On Boundedness of Q-Learning Iterates for Stochastic Shortest Path Problems

We consider a totally asynchronous stochastic approximation algorithm, Q-learning, for solving finite space stochastic shortest path (SSP) problems, which are total cost Markov decision processes with an absorbing and cost-free state. For the most commonly used SSP models, existing convergence proofs assume that the sequence of Q-learning iterates is bounded with probability one, or some other condition that guarantees boundedness. We prove that the sequence of iterates is naturally bounded with probability one, thus furnishing the boundedness condition in the convergence proof by Tsitsiklis [Tsi94] and establishing completely the convergence of Q-learning for these SSP models. Mar 2011; revised Sep 2011, Apr 2012 ∗Laboratory for Information and Decision Systems (LIDS), M.I.T. (janey [email protected]). †Laboratory for Information and Decision Systems (LIDS) and Dept. EECS, M.I.T. ([email protected]).